Robust Matrix Regression

Modern technologies are producing datasets with complex intrinsic structures, and they can be naturally represented as matrices instead of vectors. To preserve the latent data structures during processing, modern regression approaches incorporate the low-rank property to the model, and achieve satisfactory performance for certain applications. These approaches all assume that both predictors and labels for each pair of data within the training set are accurate. However, in real world applications, it is common to see the training data contaminated by noises, which can affect the robustness of these matrix regression methods. In this paper, we address this issue by introducing a novel robust matrix regression method. We also derive efficient proximal algorithms for model training. To evaluate the performance of our methods, we apply it on real world applications with comparative studies. Our method achieves the state-of-the-art performance, which shows the effectiveness and the practical value of our method. Introduction Classical regression methods, such as ridge regression (Hoerl and Kennard, 1970) and lasso (Tibshirani, 1996) are basically designed for data in vector form. However, with the development of modern technology, it is common to meet datasets with sample unit not in vector form but instead in matrix form. Examples include the two-dimensional digital images, with quantized values of different colors at certain rows and columns of pixels; and electroencephalography (EEG) data with voltage fluctuations at multiple channels over a period of time. When using traditional regression methods to process these data, we have to reshape them into vectors, which may destroy the latent topological structural information, such as the correlation between different channels for EEG data (Zhou and Li, 2014), and the spatial relation within an image (Wolf, Jhuang, and Hazan, 2007). To tackle this issue, several methods have been proposed to perform regression on data in matrix form directly. One such model is the regularized matrix regression (RGLM) (Zhou and Li, 2014). Given a dataset {Xi, yi}i=1, where N is the sample size, Xi ∈ Rp×q denotes the ith data matrix as predictor, and yi ∈ R is the corresponding output, the R-GLM model aims to learn a function f : Rp×q → R to identify the output given a newly observed data matrix with yi = tr(WXi) + b+ , (1) where tr(·) represents the trace of a matrix, W is the regression matrix with low-rank property to preserve structural information of each data matrix, and b denotes the offset. is zero-mean Gaussian noise to model the small uncertainty of the output. With this setting, the R-GLM has achieved satisfying results in several applications. However, there still exist certain issues that should be further addressed. Firstly, RGLM uses the Gaussian noise for model fitting, and take all deviations of predicted values from labels into account. This setting can be reasonable in certain cases, but may not make sense for particular applications. As an example, consider the problem of head pose estimation (Sherrah and Gong, 2001), where for each data pair, the predictor is a two dimensional digital image for the head of a person, while the output denotes the angle of his head. Because the real angle of head cannot be measured precisely, there should exist certain deviations of provided labels from the real ones empirically. In this case, the regression model should be able to tolerant such small deviations instead of taking them all into account. Another important issue is that, the predictors are also assumed to be noise free, which can be irrational in certain applications. Practically, it is common to see signals corrupted by noise, such as image signals with occlusion, specular reflections or noise (Huang, Cabral, and De la Torre, 2016), and financial data with noise (Magdon-Ismail, Nicholson, and Abu-Mostafa, 1998). Thus, it is important for a regression model to be tolerant of noise on predictors and labels to enhance its robustness empirically. In this paper, we introduce two novel matrix regression methods to tackle the above mentioned issues. We first propose a “Robust Matrix Regression” (RMR) to tackle the noisy label problem, by introducing hinge loss to model the uncertainty of regression labels. In this way, our method only considers error larger than a pre-specified value, and can tolerate error around each labeled output within a small range. This approach is also favored for other advantages in certain scenarios. As an example, in applications like financial time-series prediction, it is common to require not to lose more than money when dealing with data like exchange rates, and this issue can be well addressed with our setting. Even though the hinge loss error has been used in ar X iv :1 61 1. 04 68 6v 1 [ cs .L G ] 1 5 N ov 2 01 6 the support vector regression model (Smola and Schölkopf, 2004), it is an algorithm based on vector-form data, which can ruin the latent structure for matrix regression problem. We then propose efficient ADMM method to solve the optimization problem iteratively. To further enhance the robustness of RMR with noisy predictors, we propose a generalized RMR (G-RMR) by decomposing each data matrix as latent clean signal plus sparse outliers. For model training, we also derive a proximal algorithm to estimate both the regression matrix and latent clean signals iteratively. To evaluate the performance of our methods, we conduct extensive experiments on both approaches with comparison of state-of-the-art ones. Our methods achieve superior performance consistently, which shows their efficiency in real world problems. Notations: We present the scalar values with lower case letters (e.g., x); vectors by bold lower case letters (e.g., x); and matrix by bold upper case letters (e.g., X). For a matrix X, its (i, j)-entity is represented as Xi,j . tr(·) denotes the trace of a matrix, and {a}+ = max(0, a). We further set ||X||F and ||X||∗ as the Frobenius norm and nuclear norm of a matrix X respectively. Robust Matrix Regression We first introduce the RMR model to address the noisy label problem, with an ADMM algorithm for model training. Model For matrix regression, classical techniques need to reshape each matrix Xi into a vector xi, which will destroy its intrinsic structures, resulting in the loss of information. The R-GLM approach (Zhou and Li, 2014) addresses this issue by enforcing the regression matrix W to be low-rank representable with nuclear norm penalty. However, this method is based on the Gaussian loss, which may affect the robustness with existence of noisy labels. To tackle this issue, an intuitive idea is to ignore noises within a small margin {− , } around each label for robust model fitting. Motivated by this idea, we propose our RMR, by introducing the hinge loss for model fitting, where the residual corresponding to each data Xi is defined as follows hi(W, b) = (|tr(WXi) + b− yi| − )+. (2) With the above formulation of residuals, when learning the regression model, our approach only takes residuals larger than into account, thus, the labels contaminated by noise within a small margin is tolerable accordingly. Similar residual modeling approach has also been used in the method of support vector regression (Smola and Schölkopf, 2004). However, this approach is proposed for vector data regression and cannot capture the latent structure within each data matrix. Differently, our method can capture such latent structure by incorporating the spectral elastic net penalty (Luo et al., 2015) into the regression matrix W, which can model the correlation of each data matrix effectively. And the corresponding optimization problem is defined as follows argmin W,b H(W, b) + τ ||W||∗ (3) where H(W, b) = 1 2 tr(W>W)